perf: specialize whole-array sums for run-end arrays - #9823
Conversation
Merging this PR will degrade performance by 14.49%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
13.3 µs | 20.4 µs | -34.81% |
| ❌ | WallTime | filtered_owned_i64_avx2[OneNullInEight] |
21.9 µs | 26 µs | -15.47% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
| 🆕 | Simulation | whole_array_sum_partially_valid |
N/A | 68.7 µs | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/sum-runend (363be18) with develop (7ecae9e)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
31e3f62 to
b00acc6
Compare
|
Can we not add 35 new benchmarks maybe two or 3? |
1cba712 to
d526f93
Compare
connortsui20
changed the title
|
im just going to remove grouped aggregation, i dont really understand it myself and its making things complicated for me to understand |
31dff9b to
d8b1ebc
Compare
| //! SQL-style sums with explicit overflow and empty-input state. | ||
| //! | ||
| //! Encoding kernels produce scalar or grouped partials with the same state. Finalization returns | ||
| //! null for empty inputs and overflow, while [`Sum`] retains its legacy partial representation. |
There was a problem hiding this comment.
I changed it, I think this is useful to have (it was not obvious to me 1 week ago what the difference was, the docs on SumV2 directly are perhaps a bit too specific).
Compute primitive Sum and SumV2 from run values and lengths without fully decoding the array. Clip boundary runs to the array slice and reuse cached validity indices or scan the bounded bitmap. Grouped sums and decimals continue to use the existing fallback. Cover slice boundaries, nulls, overflow, and NaNs, and add a whole-array benchmark. Weighted floating-point sums can round differently from repeated addition; shared signed-overflow limitations remain unchanged. Signed-off-by: "Connor Tsui" <connor.tsui20@gmail.com>
d8b1ebc to
65f708b
Compare
Signed-off-by: "Connor Tsui" <connor.tsui20@gmail.com>
|
I think we should do a pass on the comments, there's some duplication across them |
8a0d897 to
363be18
Compare
3 participants
Summary
Computes whole-array
SumandSumV2directly from primitive run values and lengths, avoiding full decoding. Grouped sums and decimals use the existing fallback.Changes
Reuses cached validity indices or scans the bitmap within the array's slice. Adds coverage for slice boundaries, nulls, overflow, and NaNs, plus one whole-array benchmark. Weighted floating-point sums can round differently from repeated addition, and the shared signed-overflow limitations remain unchanged. Tests and benchmarks have not been rerun for this revision.